Depth map generation device for merging multiple depth maps

ABSTRACT

A depth map generation device for merging multiple depth maps includes at least three image capturers, a depth map generator, and a mixer. The at least three image capturers form at least two image capture pairs. The depth map generator is coupled to the at least three image capturers for generating a depth map corresponding to each image capturer pair of the at least two image capture pairs according to an image pair captured by the each image capturer. The mixer is coupled to the depth map generator for merging at least two depth maps corresponding to the at least two image capturer pairs to generate a final depth map, wherein the at least two depth maps have different characteristics.

CROSS REFERENCE TO RELATED APPLICATION

This application claims the benefit of U.S. Provisional Application No.62/479,331, filed on Mar. 31, 2017 and entitled “Camera with PanoramicImage and Depth Information and Depth Capturing Device and System,” thecontents of which are incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a depth map generation device forfusing (merging) multiple depth maps, and particularly to a depth mapgeneration device that can fuse depth maps with at least twocharacteristics.

2. Description of the Prior Art

Generally speaking, a valid area of a depth map with higher accuracy(e.g. corresponding to a long baseline) is narrower than a valid area ofa depth map with lower accuracy (e.g. corresponding to a shortbaseline), so a user may choose the depth map with lower accuracybecause of needing a larger valid area (that is, the depth map withhigher accuracy will be given up). Therefore, a depth engine provided bythe prior art can enlarge a valid area of a depth map generated by thedepth engine through a predetermined conditional judgment. For example,the depth engine can make the depth map generated by the depth enginehave lower accuracy through the predetermined conditional judgment,wherein the predetermined conditional judgment corresponds to tradeoffbetween accuracy and a range of a valid area corresponding to a depthmap. That is, the depth engine can either make the depth map generatedby the depth engine have larger valid area (but have lower accuracy), ormake the depth map generated by the depth engine have smaller valid area(but have higher accuracy) through the predetermined conditionaljudgment. That is, the depth engine cannot generate a depth mapsimultaneously with different characteristics (e.g. accuracy and a rangeof a valid area) through the predetermined conditional judgment.Therefore, the prior art is not a good technical solution for the user.

SUMMARY OF THE INVENTION

An embodiment of the present invention provides a depth map generationdevice for merging multiple depth maps. The depth map generation deviceincludes at least three image capturers, a depth map generator, and amixer. The at least three image capturers is used for forming at leasttwo image capture pairs. The depth map generator is coupled to the atleast three image capturers for generating a depth map corresponding toeach image capturer pair of the at least two image capture pairsaccording to an image pair captured by the each image capturer pair. Themixer is coupled to the depth map generator for merging at least twodepth maps corresponding to the at least two image capture pairs togenerate a final depth map, wherein the at least two depth maps havedifferent characteristics.

Another embodiment of the present invention provides a depth mapgeneration device for merging multiple depth maps. The depth mapgeneration device includes at least two image capturers, a light source,a depth map generator, and a mixer. The light source is used foremitting structured light, wherein the light source and the at least twoimage capturers form at least two image capture pairs, respectively. Thedepth map generator is coupled to the at least two image capturers forgenerating a depth map corresponding to each image capturer pair of theat least two image capture pairs according to an image comprising thestructured light captured by the each image capturer pair. The mixer iscoupled to the depth map generator for merging at least two depth mapscorresponding to the at least two image capture pairs to generate afinal depth map, wherein the at least two depth maps have differentcharacteristics.

Another embodiment of the present invention provides a depth mapgeneration device for merging multiple depth maps. The depth mapgeneration device includes an image capture module, a depth mapgenerator, and a mixer. The image capture module includes at least twoimage capture pairs, and each image capturer pair of the at least twoimage capture pairs is composed of two image capturers, or composed ofan image capturer and a light source. The depth map generator is coupledto the image capture module for generating at least two depth mapscorresponding to the at least two image capture pairs according toimages captured by the at least two image capture pairs. The mixer iscoupled to the depth map generator for merging the at least two depthmaps to generate a final depth map, wherein the at least two depth mapshave different characteristics.

The present invention provides a depth map generation device for fusing(merging) multiple depth maps. The depth map generation device utilizesat least three image capturers (or at least two image capturers and alight source) to generate at least two depth maps corresponding to atleast two baselines, wherein the at least two depth maps correspondingto the at least two baselines have at least two characteristics. Then,the present invention can generate a final depth map by fusing the atleast two depth maps with the at least two characteristics. Therefore,compared to the prior art, the present invention can enlarge a range ofthe final depth map, or increase accuracy of the final depth map

These and other objectives of the present invention will no doubt becomeobvious to those of ordinary skill in the art after reading thefollowing detailed description of the preferred embodiment that isillustrated in the various figures and drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating a depth map generation device formerging multiple depth maps according to a first embodiment of thepresent invention.

FIG. 2 is a diagram illustrating effective ranges of depth mapscorresponding to two different resolutions.

FIG. 3 is a diagram illustrating the first baseline and the secondbaseline not parallel to each other.

FIG. 4 is a diagram illustrating the depth map generation device furtherincluding a light source.

FIG. 5A is a diagram illustrating the first coding pattern.

FIG. 5B is a diagram illustrating the second coding pattern.

FIG. 6 is a diagram illustrating the third coding pattern.

FIG. 7 is a diagram illustrating a depth map generation device formerging multiple depth maps according to a second embodiment of thepresent invention.

FIG. 8 is a diagram illustrating a depth map generation device formerging multiple depth maps according to a third embodiment of thepresent invention.

DETAILED DESCRIPTION

Please refer to FIGS. 1, 2. FIG. 1 is a diagram illustrating a depth mapgeneration device 100 for merging multiple depth maps according to afirst embodiment of the present invention, and FIG. 2 is a diagramillustrating effective ranges of depth maps corresponding to twodifferent resolutions. As shown in FIG. 1, the depth map generationdevice 100 includes three image capturers 102, 104, 106, a depth mapgenerator 108, and a mixer 110. But, the present invention is notlimited to the depth map generation device 100 only including the threeimage capturers 102, 104, 106. That is, the depth map generation device100 can include more than four image capturers. As shown in FIG. 1, afirst baseline BL1 (e.g. 12 cm) exists between the image capturer 102and the image capturer 106 and a second baseline BL2 (e.g. 3 cm) existsbetween the image capturer 102 and the image capturer 104, wherein theimage capturers 102, 104, 106, the depth map generator 108, and themixer 110 are installed on a printed circuit board 112. But, forsimplifying FIG. 1, only the image capturers 102, 104, 106 are shown onthe printed circuit board 112.

The image capturers 102, 104, 106 are used for forming two image capturepairs, wherein the image capturers 102, 106 form a first image capturepair and the image capturers 102, 104 form a second image capture pair.As shown in FIG. 1, the depth map generator 108 electrically connectedto the image capturers 102, 104, 106 is used for generating a depth mapcorresponding to each image capturer pair of the first image capturepair and the second image capture pair according to an image paircaptured by the each image capturer pair. That is, the depth mapgenerator 108 can generate two depth maps DP1, DP2 corresponding to thefirst image capture pair and the second image capture pair,respectively. As shown in FIG. 2, because a length of the first baselineBL1 is greater than a length of the second baseline BL2, accuracy of thedepth map DP1 is higher than accuracy of the depth map DP2, but a validarea of the depth map DP1 is narrower than a valid area of the depth mapDP2. That is, a distance D1 corresponding to a maximum disparity of thedepth map DP1 is greater than a distance D2 corresponding to a maximumdisparity of the depth map DP2.

After the depth map generator 108 generates the depth maps DP1, DP2corresponding to the first image capture pair and the second imagecapture pair, the mixer 110 electrically connected to the depth mapgenerator 108 can fuse (merge) the depth maps DP1, DP2 to generate afinal depth map FIDP according to a first rule. The first rule includesan invalid area IVA1 (corresponding to the distance D1) of the depth mapDP1 corresponding to the first baseline BL1 being replaced with a validarea VA2 of the depth map DP2 (as shown in FIG. 2) corresponding to thesecond baseline BL2. In addition, an invalid area of a depth map of thepresent invention can be determined by factors of view angle, baseline,and so on. For example, a disparity d corresponding to the depth map DP1can be determined according to equation (1):d=RES*BL/(Z*2 tan(FOV/2))  (1)

As shown in equation (1), RES is resolution of the depth map DP1, BL isa baseline (that is, the first baseline BL1) between the image capturers102, 106, FOV is an view angle of the image capturer 102, 106, and Z isa distance within the depth map DP1 corresponding to an object (that is,a depth within the depth map DP1 corresponding to the object). That is,substituting the maximum disparity corresponding to the depth map DP1,the resolution of the depth map DP1, the view angle of the imagecapturer 102, and the baseline (that is, the first baseline BL1) betweenthe image capturers 102, 106 into equation (1) can yield a depth (thatis, the distance D1) of the invalid area IVA1 corresponding to the depthmap DP1.

But, in another embodiment of the present invention, the first ruleincludes the invalid area IVA1 of the depth map DP1 and a predeterminedcontiguous area within the depth map DP1 adjacent to the invalid areaIVA1 are replaced with a corresponding valid area of the depth map DP2to prevent the mixer 110 from generating the final depth map FIDPincluding a part of the invalid area IVA1 of the depth map DP1 becausean error of the invalid area IVA1 of the depth map DP1, wherein a rangeof the predetermined contiguous area can be adjusted according to designrequirements or usage scenarios. In addition, because the length of thefirst baseline BL1 (e.g. 12 cm) is greater than the length of the secondbaseline BL2 (e.g. 3 cm), when the depth maps DP1, DP2 are representedby disparity, normalization needs to be executed on disparity of thevalid area VA2 of the depth map DP2, that is, the disparity of the validarea VA2 of the depth map DP2 needs to be multiplied by a normalizationratio to match disparity of the depth map DP1, wherein the normalizationratio is determined by equation (2):NRA=BL1/BL2  (2)

As shown in equation (2), NRA is the normalization ratio, BL1 is thelength of the first baseline BL1, and BL2 is the length of the secondbaseline BL2. Therefore, the final depth map FIDP will include the validarea VA1 of the depth map DP1 and the valid area VA2 of the depth mapDP2, wherein the disparity of the valid area VA2 is a normalizeddisparity.

In addition, when the first baseline BL1 and the second baseline BL2 arenot parallel to each other (as shown in FIG. 3), the first rule furtherincludes geometric calibration. In one embodiment of the presentinvention, the geometric calibration is rotation calibration, and themixer 110 can utilize a rotation matrix to execute geometric conversionon at least one of the depth maps DP1, DP2 to make the first baselineBL1 and the second baseline BL2 parallel to each other. For example, themixer 110 can utilize a first rotation matrix to rotate the depth mapDP1, or can also utilize the first rotation matrix and a second rotationmatrix to rotate the depth maps DP1, DP2, respectively. The firstrotation matrix and the second rotation matrix correspond to one of thefirst baseline BL1 and the second baseline BL2, or correspond to areference line different from the first baseline BL1 and the secondbaseline BL2.

In addition, in another embodiment of the present invention, when thedepth map DP1 and the depth map DP2 are represented by distance,although the accuracy of the depth map DP1 and the accuracy of the depthmap DP2 are different (the accuracy of the depth map DP1 is greater thanthe accuracy of the depth map DP2), because a unit of the distance isidentical (e.g. meter), any normalization conversion is not executed onthe depth map DP2 to make the depth map DP2 match the depth map DP1.

In addition, in another embodiment of the present invention, the mixer110 generates the final depth map FIDP according to smoothness of thedepth map DP1 and the depth map DP2. For example, in one embodiment ofthe present invention, when smoothness of edges of a first block of thedepth map DP1 is better than smoothness of edges of a firstcorresponding block of the depth map DP2, the final depth map FIDPgenerated by the mixer 110 will include the first block of the depth mapDP1, wherein the first block includes at least one pixel; whensmoothness of edges of a second block of the depth map DP1 is worse thansmoothness of edges of a second corresponding block of the depth mapDP2, the final depth map FIDP generated by the mixer 110 will includethe second corresponding block of the depth map DP2, wherein the secondblock also includes at least one pixel. In addition, the mixer 110 cancompare the smoothness of the depth map DP1 with the smoothness of thedepth map DP2 after or before the normalization is executed on the depthmap DP2. In addition, in another embodiment of the present invention,after the normalization is executed on the disparity of the depth mapDP2, when a difference between an average depth corresponding to a thirdblock of the depth map DP1 and an average depth corresponding to a thirdcorresponding block of the depth map DP2 is greater than a predeterminedvalue, the final depth map FIDP generated by the mixer 110 will includethe third corresponding block of the depth map DP2 (because when thedifference between the average depth corresponding to the third block ofthe depth map DP1 and the average depth corresponding to the thirdcorresponding block of the depth map DP2 is greater than thepredetermined value, it means that the third block of the depth map DP1is located at the invalid area IVA1 of the depth map DP1), wherein thethird block and the third corresponding block include at least onepixel; when a difference between an average depth corresponding to afourth block of the depth map DP1 and an average depth corresponding toa fourth corresponding block of the depth map DP2 is less than thepredetermined value, the final depth map FIDP generated by the mixer 110will include the fourth block of the depth map DP1 (because the accuracyof the depth map DP1 is higher than the accuracy of the depth map DP2),wherein the fourth block and the fourth corresponding block also includeat least one pixel. In addition, in another embodiment of the presentinvention, the mixer 110 generates the final depth map FIDP according toat least one of smoothness of each block of the depth map DP1 andsmoothness of each block of the depth map DP2, an average depth of eachblock of the depth map DP1 and an average depth of each block of thedepth map DP2, and valid area/invalid area of the depth map DP1 andvalid area/invalid area of the depth map DP2.

In addition, the depth map generator 108 can be a field programmablegate array (FPGA) with the above mentioned functions of the depth mapgenerator 108, or an application-specific integrated circuit (ASIC) withthe above mentioned functions of the depth map generator 108, or asoftware module with the above mentioned functions of the depth mapgenerator 108. In addition, the mixer 110 can be a field programmablegate array with the above mentioned functions of the mixer 110, or anapplication-specific integrated circuit with the above mentionedfunctions of the mixer 110, or a software module with the abovementioned functions of the mixer 110. In addition, in another embodimentof the present invention, the depth map generator 108 and the mixer 110can be integrated into a first processor, wherein the first processorcan be a field programmable gate array with the above mentionedfunctions of the depth map generator 108 and the mixer 110, or anapplication-specific integrated circuit with the above mentionedfunctions of the depth map generator 108 and the mixer 110.

In addition, when the depth map generation device 100 includes at leastfour image capturers (can format least three image capture pairs), thefirst rule includes an invalid area of a depth map corresponding to anN^(th) baseline being replaced with a valid area of a depth mapcorresponding to an (N+1)^(th) baseline, and disparity of the depth mapcorresponding to the N^(th) baseline is multiplied by an N^(th)normalization ratio and disparity of the depth map corresponding to the(N+1)^(th) baseline is multiplied by an (N+1)^(th) normalization ratioto make normalized disparity of the depth map corresponding to theN^(th) baseline and normalized disparity of the depth map correspondingto the (N+1)^(th) baseline match disparity of a depth map correspondingto the first baseline BL1, wherein the length of the first baseline BL1is greater than lengths of other baselines of baselines corresponding tothe at least three image capture pairs different from the first baselineBL1, a length of the N^(th) baseline is greater than a length of the(N+1)^(th) baseline, and N is a positive integer.

Please refer to FIG. 4. FIG. 4 is a diagram illustrating the depth mapgeneration device further including a light source. As shown in FIG. 4,the depth map generation device 100 includes the image capturers 102,104, 106, a light source 702, the depth map generator 108, and the mixer110, wherein in one embodiment of the present invention, the lightsource 702 can be an infrared light source for emitting structured light(or a random pattern), and a function of the light source 702 is usedfor making better quality of the depth maps DP1, DP2 generated by thedepth map generation device 100. But, the present invention is notlimited to light source 702 being an infrared light source. That is, thelight source 702 can be other type of light sources (e.g. the lightsource 702 can be a visible light source). Or, in another embodiment ofthe present invention, the depth map generation device 100 can alsoinclude at least one infrared laser light source. Taking the depth mapgeneration device 100 as an example, the light source 702 is turned onaccording to at least one of luminance of an environment which the depthmap generation device 100 is located at, the quality of the depth mapDP1 (or the depth map DP2, and a difference of the depth map DP1 (or thedepth map DP2) corresponding to turning-on and turning-off of the lightsource 702.

When the light source 702 is turned on according to the luminance of theenvironment which the depth map generation device 100 is located at, acontroller (not shown in FIG. 4) can determine the luminance of theenvironment which the depth map generation device 100 is located ataccording to at least one of a shutter time, an exposure time, and anISO gain currently set by an image capturer (e.g. the image capturer102, the image capturer 104, or the image capturer 106). Taking theexposure time as an example, in one embodiment of the present invention,when the shutter time of the image capturer 102 is fixed (or the imagecapturer 102 has no shutter), the controller can determine whether toturn on the light source 702 according to a value GEX generated byequation (3):GEX=gain*EXPT  (3)

As shown in equation (3), “gain” shown in equation (3) is the ISO gaincorresponding to the image capturer 102 and “EXPT” shown in equation (3)is the exposure time corresponding to the image capturer 102. When thevalue GEX is greater than a high threshold value, it means that theluminance of the environment which the depth map generation device 100is located at is too dark, so the controller turns on the light source702; and when the value GEX is less than a low threshold value, it meansthat the luminance of the environment which the depth map generationdevice 100 is located at is bright enough, so the controller turns offthe light source 702, wherein the high threshold value is greater thanthe low threshold value. In addition, when a maximum value of the valueGEX (corresponding to a maximum exposure time and a maximum gain of theimage capturer 102) cannot be always greater than the high thresholdvalue, the controller can turn on the light source 702 according tocurrent luminance of the environment which the depth map generationdevice 100 is located at.

When the light source 702 is turned on according to the quality of thedepth map DP1, the controller can determine the quality of the depth mapDP1 according to at least one of a number of pixels with invalid valueswithin the depth map DP1 and smoothness of the depth map DP1. Forexample, in one embodiment of the present invention, the controller candetermine whether to turn on the light source 702 according to a valueCOST generated by equation (4):COST=a*mean(HPF(x))+b*invalid_cnt(x)  (4)

As shown in equation (4), “HPF(x)” shown in equation (4) corresponds toa response of a high pass filter (because the smoothness of the depthmap DP1 corresponds to high frequency areas of the depth map DP1),“mean(HPF(x))” shown in equation (4) corresponds to an average of theresponse of the high pass filter (but, in another embodiment of thepresent invention, “mean (HPF(x))” shown in equation (4) can be replacedwith a sum corresponding to the response of the high pass filter),“invalid_cnt(x)” shown in equation (4) represents the number of thepixels of with the invalid values, “x” shown in equation (4) representsthe depth map DP1, and “a, b” shown in equation (4) are coefficients.When the value COST is greater than a threshold value, it means that theluminance of the environment which the depth map generation device 100is located at is too dark or shot objects of the depth map DP1 have notexture, so the controller turns on the light source 702. In addition,after the light source 702 is turned on for a predetermined time, thecontroller can attempt to turn off the light source 702 and make theimage capturer 102 capture at least one image, and then the controllercalculates a cost value corresponding to the at least one imageaccording to equation (4). If the cost value corresponding to the atleast one image is still greater than the threshold value, thecontroller turns on the light source 702 again and executes the abovementioned operation again after the controller turns on the light source702 for the predetermined time; and if the cost value corresponding tothe at least one image is less than the threshold value, the controllerturns off the light source 702 until the cost value corresponding to theat least one image is greater than the threshold value again.

In addition, the controller can turn on and turn off the light source702, and determine the quality of the depth map DP1 according to thedifference of the depth map DP1 corresponding to turning-on andturning-off of the light source 702. If the difference of the depth mapDP1 corresponding to turning-on and turning-off of the light source 702is less than a reference value, it means that turning-on and turning-offof the light source 702 does not influence the quality of the depth mapDP1, so the controller can turn off the light source 702.

In addition, after the light source 702 is turned on, the controller canoptionally adjust intensity of the light source 702 according toluminance corresponding to a plurality of images captured by the imagecapturers 102, 104, 106 and a target value, wherein the target value isset according to reflection coefficient of a human skin of a usercorresponding to the structured light emitted by the light source 702.For example, the controller can generate a luminance distribution mapcorresponding to the plurality of images according to the plurality ofimages, and optionally adjust the intensity of the light source 702according to a percentage of the depth map DP1 occupied by an areacorresponding to a maximum luminance value of at least one luminancevalue within the luminance distribution map greater than the targetvalue. In addition, in another embodiment of the present invention, thecontroller can generate average luminance corresponding to the pluralityof images according to the plurality of images, and optionally adjustthe intensity of the light source 702 according to the average luminanceand the target value. In addition, in another embodiment of the presentinvention, the controller can generate a luminance histogramcorresponding to a plurality of pixels of the plurality of imagesaccording to the plurality of images, and optionally adjust theintensity of the light source 702 according to a median of the luminancehistogram and the target value, or according to a quartile of theluminance histogram and the target value.

In addition, in another embodiment of the present invention, after thelight source 702 is turned on, the controller can optionally dynamicallyadjust the intensity of the light source 702 according to a distancebetween at least one predetermined object within the plurality of imagesand the image capturer (e.g. the image capturer 102, the image capturer104, or the image capturer 106) and a first lookup table, wherein thefirst lookup table stores relationships between a distance correspondingto an object and the intensity of the light source 702. In addition, inanother embodiment of the present invention, the controller canoptionally dynamically adjust the intensity of the light source 702according to the distance between the at least one predetermined objectwithin the plurality of images and the image capturer (e.g. the imagecapturer 102, the image capturer 104, or the image capturer 106) and afirst correlation formula.

In addition, in another embodiment of the present invention, thecontroller continuously detects the luminance of the environment whichthe depth map generation device 100 is located at under the light source702 being turned off. When the luminance of the environment is brighter,the controller increases the intensity of the light source 702 (when thelight source 702 is turned on) according to a second lookup table,wherein the second lookup table stores relationships between theintensity of the light source 702 (when the light source 702 is turnedon) and the luminance of the environment. In addition, in anotherembodiment of the present invention, when the luminance of theenvironment is brighter, the controller increases the intensity of thelight source 702 (when the light source 702 is turned on) according to asecond correlation formula.

In addition, in another embodiment of the present invention, after thecontroller first turns off the light source 702, the controller detectsthe luminance of the environment. Then, according to an automaticexposure (AE) algorithm well-known to one of ordinary skill in the art,the controller utilizes the exposure time (or at least one of theshutter time, the exposure time, and the ISO gain) of the image capturer(e.g. the image capturer 102, the image capturer 104, or the imagecapturer 106) to make the luminance of the environment be reduced to notto interfere with the image capturer (e.g. the image capturer 102, theimage capturer 104, or the image capturer 106), and fixes a currentexposure time of the image capturer (e.g. the image capturer 102, theimage capturer 104, or the image capturer 106). Then, the controllerturns on the light source 702 and detects the intensity of the lightsource 702 until the intensity of the light source 702 is up to thetarget value.

In addition, in one embodiment of the present invention, the structuredlight provided by the light source 702 is a coding pattern(corresponding to a random pattern). But, in another embodiment of thepresent invention, the structured light can combine a first codingpattern with a second coding pattern. Please refer to FIGS. 5A, 5B. FIG.5A is a diagram illustrating the first coding pattern, and FIG. 5B is adiagram illustrating the second coding pattern. As shown in FIG. 5A, thefirst coding pattern is divided into a plurality of blocks according tothe second coding pattern, and then the second coding pattern is appliedto the plurality of blocks to form the structured light. For example, asshown in FIG. 5B, when the second coding pattern is applied to a block402 of the plurality of blocks, luminance of pixels within the block 402needs to be reduced. That is, only luminance of pixels within the block402 being turned on needs to be reduced, and other pixels within theblock 402 not being turned on need to be turned off continuously. Inaddition, as shown in FIG. 5B, when the second coding pattern is appliedto a block 404 of the plurality of blocks, luminance of pixels withinthe block 404 needs to be increased. That is, only luminance of pixelswithin the block 404 being turned on needs to be increased, and otherpixels within the block 404 not being turned on need to be turned offcontinuously to form the structured light. In addition, in anotherembodiment of the present invention, when the second coding pattern isapplied to the plurality of blocks, luminance of pixels within eachblock of the plurality of blocks being turned on can have multi-levelchanges. In addition, in another embodiment of the present invention, athird coding pattern shown in FIG. 6 can be also applied to the firstcoding pattern (as shown in FIG. 5A) to form the structured light.

Please refer to FIG. 7. FIG. 7 is a diagram illustrating a depth mapgeneration device 500 for merging multiple depth maps according to asecond embodiment of the present invention, wherein the depth mapgeneration device 500 includes two image capturers 104, 106, a lightsource 502, a depth map generator 108, and a mixer 110. But, the presentinvention is not limited to the depth map generation device 500 onlyincluding the two image capturers 104, 106. That is, the depth mapgeneration device 500 can include more than three image capturers. Inaddition, the light source 502 and the image capturers 104, 106 form twoimage capture pairs respectively, wherein the light source 502 and theimage capturer 106 form a first image capture pair and the light source502 and the image capturer 104 form a second image capture pair. Asshown in FIG. 7, a first baseline BL1 (e.g. 12 cm) exists between thelight source 502 and the image capturer 106 and a second baseline BL2(e.g. 3 cm) exists between the light source 502 and the image capturer104. As shown in FIG. 7, the depth map generator 108 is electricallyconnected to the image capturers 104, 106 for generating a depth mapcorresponding to each image capturer pair of the first image capturepair and the second image capture pair according to an image includingthe structured light captured by the each image capturer pair. That is,the depth map generator 108 will generate two depth maps correspondingto the first image capture pair and the second image capture pair(wherein effective ranges of the two depth maps corresponding to thefirst image capture pair and the second image capture pair can bereferred to FIG. 2). In addition, operational principles of the depthmap generator 108 and the mixer 110 of the depth map generation device500 can be referred to operational principles of the depth map generator108 and the mixer 110 of the depth map generation device 100, so furtherdescription thereof is omitted for simplicity. In addition, operationalprinciples of the light source 502 can also be referred to operationalprinciples of the light source 702, so further description thereof isalso omitted for simplicity.

In addition, please refer to FIG. 8. FIG. 8 is a diagram illustrating adepth map generation device 800 for merging multiple depth mapsaccording to a third embodiment of the present invention, wherein thedepth map generation device 800 includes image capturers 102, 104, 106,a light source 602, a depth map generator 108, and a mixer 110, whereinthe depth map generator 108 is coupled to the image capturers 102, 104,106, and the image capturers 102, 104, 106 and the light source 602 forman image capture module. But, in another embodiment of the presentinvention, the depth map generation device 800 can include at leastthree image capturers and at least one light source. As shown in FIG. 8,the image capturers 102, 104, 106 form two first image capture pairs,and the light source 602 and the image capturers 102, 104, 106 formthree second image capture pairs respectively, wherein the three secondimage capture pairs are used for capturing images including structuredlight generated by the light source 602. Therefore, the depth mapgenerator 108 can be used for generating at least two depth mapscorresponding to at least two image capture pairs of the two first imagecapture pairs and the three second image capture pairs according toimages captured by the at least two image capture pairs. In addition,operational principles of the three second image capture pairs can bereferred to operational principles of the first image capture pair(composed of the light source 502 and the image capturer 106) of thedepth map generation device 500 and the second image capture pair(composed of the light source 502 and the image capturer 104) of thedepth map generation device 500 shown in FIG. 7, so further descriptionthereof is omitted for simplicity. In addition, operational principlesof the two first image capture pairs can be referred to operationalprinciples of the first image capture pair (composed of the imagecapturers 102, 106) of the depth map generation device 100 and thesecond image capture pair (composed of the image capturers 102, 104) ofthe depth map generation device 100, so further description thereof isomitted for simplicity. In addition, subsequent operational principlesof the depth map generation device 800 can be referred to theoperational principles of the depth map generation devices 500, 100, sofurther description thereof is omitted for simplicity.

To sum up, the depth map generation device provided by the presentinvention utilizes at least three image capturers (or at least two imagecapturers and a light source) to generate at least two depth mapscorresponding to at least two baselines, wherein the at least two depthmaps corresponding to the at least two baselines have at least twocharacteristics. Then, the present invention can generate a final depthmap according to the at least two depth maps with the at least twocharacteristics. Therefore, compared to the prior art, the presentinvention can enlarge a range of the final depth map, or increaseaccuracy of the final depth map.

Those skilled in the art will readily observe that numerousmodifications and alterations of the device and method may be made whileretaining the teachings of the invention. Accordingly, the abovedisclosure should be construed as limited only by the metes and boundsof the appended claims.

What is claimed is:
 1. A depth map generation device for mergingmultiple depth maps, comprising: at least three image capturers formingat least two image capture pairs; a depth map generation circuit coupledto the at least three image capturers for generating a depth mapcorresponding to each image capturer pair of the at least two imagecapture pairs according to an image pair captured by the each imagecapturer pair; and a mixing circuit coupled to the depth map generationcircuit for merging at least two depth maps corresponding to the atleast two image capture pairs to generate a final depth map, wherein theat least two depth maps have different characteristics; wherein when themixing circuit merges the at least two depth maps to generate the finaldepth map, the mixing circuit executes normalization on at least one ofthe at least two depth maps to make disparities of the at least twodepth maps match each other.
 2. The depth map generation device of claim1, wherein the each image capturer pair has a baseline, and the at leasttwo depth maps correspond to at least two baselines.
 3. The depth mapgeneration device of claim 2, wherein the mixing circuit replaces aninvalid area of a first depth map of the at least two depth maps with acorresponding valid area of a second depth map of the at least two depthmaps to generate the final depth map.
 4. The depth map generation deviceof claim 1, wherein after the normalization is executed on the seconddepth map, disparity of the corresponding valid area of the second depthmap is multiplied by a normalization ratio, and the normalization ratiois equal to a length of a first baseline corresponding to the firstdepth map being divided by a length of a second baseline correspondingto the second depth map.
 5. The depth map generation device of claim 1,wherein when the at least two baselines are not parallel to each other,the mixing circuit further executes geometric calibration on at leastone of the at least two depth maps.
 6. The depth map generation deviceof claim 5, wherein the geometric calibration is rotation calibration.7. The depth map generation device of claim 1, wherein the mixingcircuit merges a first depth map of the at least two depth maps and asecond depth map of the at least two depth maps to generate the finaldepth map according to smoothness of each block of the first depth mapand smoothness of a corresponding block of the second depth map, andboth the each block and the corresponding block comprise at least onepixel.
 8. The depth map generation device of claim 1, furthercomprising: at least one light source, wherein each light source of theat least one light source is used for emitting structured light, and thestructured light comprises at least two coding patterns.
 9. A depth mapgeneration device for merging multiple depth maps, comprising: at leastthree image capturers forming at least two image capture pairs; a depthmap generation circuit coupled to the at least three image capturers forgenerating a depth map corresponding to each image capturer pair of theat least two image capture pairs according to an image pair captured bythe each image capturer pair; and a mixing circuit coupled to the depthmap generation circuit for merging at least two depth maps correspondingto the at least two image capture pairs to generate a final depth mapaccording to smoothness of each block of the at least two depth maps,wherein the each block comprises at least one pixel.
 10. A depth mapgeneration device for merging multiple depth maps, comprising: an imagecapture module comprising at least two image capture pairs, and eachimage capturer pair of the at least two image capture pairs is composedof two image capturers, or composed of an image capturer and a lightsource; a depth map generation circuit coupled to the image capturemodule for generating at least two depth maps corresponding to the atleast two image capture pairs according to images captured by the atleast two image capture pairs; and a mixing circuit coupled to the depthmap generation circuit for merging the at least two depth maps togenerate a final depth map according to smoothness of each block of theat least two depth maps, wherein the each block comprises at least onepixel, and the at least two depth maps have different characteristics.11. The depth map generation device of claim 10, wherein the lightsource is further used for emitting structured light, and when the eachimage capturer pair is composed of the image capturer and the lightsource, the each image capturer pair is used for capturing imagescomprising the structured light.
 12. The depth map generation device ofclaim 10 wherein the at least two image capture pairs comprise a firstimage capture pair and a second image capture pair, and when both thefirst image capture pair and the second image capture pair are composedof two image capturers, the first image capture pair and the secondimage capture pair share an image capturer comprised in the second imagecapture pair.
 13. The depth map generation device of claim 10, whereinthe at least two image capture pairs comprise a first image capture pairand a second image capture pair, and when the first image capture pairis composed of two image capturers and the second image capture pair iscomposed of an image capturer and the light source, one image capturercomprised in the first image capture pair is the image capturercomprised in the second image capture pair.